Conditional gradient type methods for composite nonlinear and stochastic optimization

In this paper, we present a conditional gradient type (CGT) method for solving a class of composite optimization problems where the objective function consists of a (weakly) smooth term and a strongly convex term. While including this strongly convex term in the subproblems of the classical conditional gradient (CG) method improves its convergence rate for solving strongly convex problems, it does not cost per iteration as much as general proximal type algorithms. Moreover, we show that existence of this strongly convex term enables us to establish convergence properties of the CGT method when the (weakly) smooth term is possibly nonconvex. In particular, we present a unified analysis for the CGT method in the sense that it achieves the best known convergence rate when the weakly smooth term is nonconvex and possesses (nearly) optimal complexity if it turns out to be convex (and hence the problem is strongly convex). While implementation of the CGT method requires explicitly estimating problem parameters like the level of smoothness for the weakly smooth term, we also present a few variants of this method which relax such estimation. Unlike general proximal type parameter free methods, these variants of the CGT method do not require any additional effort for computing (sub)gradients of the objective function and/or solving extra subproblems in each iteration. We then generalize these methods under stochastic setting and present a few new complexity results. To the best of our knowledge, this is the first time that such complexity results are presented for solving both weakly smooth nonconvex and strongly convex stochastic optimization problems. keywords iteration complexity, nonconvex optimization, strongly convex optimization, conditional gradient type methods, unified methods, weakly smooth functions